Swinging in the Archives: In-House File Transfer of DATs

Not long ago Digital Audio Tapes (DATs) were a popular medium for professional digital recording. The Smithsonian Institution Archives holds almost 2,000 DATs in its accessions but until recently lacked equipment to retrieve the rich collections of music, interviews, and programs from across the Institution. Very few archives have conducted in-house transfers of DATs. The Smithsonian Institution Archives launched a successful pilot in 2010 to transfer the audio from the DATs to its secure servers and LTO tapes. These DATs contain priceless material from Smithsonian Jazz Masterworks Orchestra concerts to Native American radio programs to events at the Freer-Sackler Galleries of Art. This project is making these recordings accessible for the long term, as DAT machines are hard to acquire. The audio is being saved in the WAV/BWF preservation formats. Best practice procedures include checksums, quality assurance reviews, and regular backups. This paper discusses how the pilot operated and the challenges SIA encountered. Introduction The Smithsonian Institution Archives (SIA) is responsible for preserving the institutional memory of a unique cultural, historical, and scientific complex of 19 museums and galleries, nine research centers, and the National Zoo. SIA’s collections contain everything from handwritten correspondence to 19century images to 21-century social media accounts and websites. These collections include nearly 2,000 Digital Audio Tapes (DATs). The National Museum of the American Indian, the FreerSackler Art Galleries, the Hirshhorn and Sculpture Garden, and Smithsonian Productions have all transferred these tapes to the Archives, contributing to Smithsonian’s permanent record. In addition to DATs (see figure 1), SIA has audio on wax cylinders, aluminum discs, LPs, reel-to-reel tapes, cassettes, microcassettes, CDs, and DVDs. Digitization of other audio recordings has been conducted internally and externally over the years. This paper documents what SIA started doing with its collection of DATs. DATs background Developed by Sony, Digital Audio Tapes were introduced in 1987 and were touted as having the same quality as CDs with recording ability [1]. “Digital audio tape (DAT) promises to be the music medium of the future. It renders sound quality on a par with digital compact discs, but in a smaller package. Moreover, DAT provides the opportunity of digital recording to hi-fi enthusiasts – a feature which the playback-only CD lacks. ... Indeed, DAT can sound splendid – but it doesn’t make bit-for-bit clones of CDs any more, than today’s conventional analog cassette decks do,” Popular Mechanics wrote in 1987 [2]. Legal issues arose over copyright, resulting in a drive for anti-copying mechanisms and a “DAT tax” before DAT recorders could be sold in the United States in 1990. Sony announced in November 2005 that it was ending production of DAT machines, making this an obsolete technology [3]. Figure 1. Digital Audio Tape or DAT recorded in 1990 for the Folk Masters radio program. SIA Accession 06-106, Smithsonian Press/Smithsonian Productions, Productions, 1991-1997. DATs never really caught on with the general consumer market due to other emerging technologies and price, meaning fewer machines were sold, and resulting in fewer technicians who can repair DAT recorders. The DAT format also was used for data storage as Digital Data Storage (DDS) tapes [4]. Blank DATs and used DAT machines continue to be available for sale online. DATs are about half the size of an analog cassette. The tape is contained inside the plastic housing with a slider that retracts, just like a VCR tape. DAT recorders rely on a rotary head, again using videotape technologies. Tapes are between 15 and 180 minutes in length and are recorded on one side. DATs are uncompressed, and the recordings have no tape hiss. Sampling frequencies are 32, 44.1, or 48 kHz at 16-bit linear quantization, while 96 kHz sampling frequency was available in some machines [5, 6]. Compared to open reel or analog cassette equipment, DAT recorders are complex and known for mechanical misalignment problems. “DAT machines used in professional settings were typically worked hard, and the heads are likely to be worn from extensive use. As fewer and fewer machines become usable over time, archives must determine whether they have enough head life on their machines to transfer their holdings,” according to The Field Audio Collection Evaluation Tool (FACET) report from the Archives of Traditional Music at Indiana University [7]. DATs also are susceptible to fungus on the tape due to storage at high temperatures/humidity. Other issues include dirt, oxide loss, and “clumps of coating material become trapped in the head gap or debris adheres to the head leading to drop-outs or worse.” [8] The Association for Recorded Sound Collections Technical Committee’s recommendations on archival sound recordings from 2009 state that DATs are inappropriate for archival storage due to “dropouts and mechanical problems” and because the machines were discontinued [9]. These issues stress that thousands of hours of Smithsonian recordings are at risk. SIA pilot SIA has been receiving DATs for nearly 10 years but lacked equipment to listen to or transfer the audio. Smithsonian Folkways [10] had a DAT player available for loan. SIA decided to pilot a DAT-to-computer transfer project with a 2010 full-time summer intern. Research focused on DATs and DAT machines, other audio preservation projects, and software options, while also testing/validating workflow and procedures for transfer. Most preservation literature refers to analog-to-digital rather than digital-to-digital preservation audio projects. Research found very few organizations doing their own in-house DAT transfers, which included the National Security Archive and Democracy Now [11]. External audio preservation experts and Smithsonian sound production staff also were consulted. Equipment includes a Mac Pro 4.1/OS 10.6.3, Audacity 1.2.5 software, a Panasonic Professional Digital Audio Tape Recorder SV-3800, and AKG K240 Monitor stereo headphones. The Mac and DAT recorder are connected through the AES/EBU IEC Type II optical output from the DAT machine to the Mac’s S/PDIF optical input. The digital-to-digital transfers are conducted in SIA’s Audiovisual Preservation Lab, where videotape digitization is also done. The audio files are exported as WAV or Waveform Audio File Format using Audacity. This is SIA’s audio preservation format because it is uncompressed, works in Windows, Mac, and Linux environments, and has been widely used in cultural heritage institutions such as the Library of Congress, Libraries and Archives Canada, and the Florida Digital Archive [12]. SIA’s best practices include using formats that possess accessibility and longevity functions such as WAV for audio, TIF for images, and PDF for text. BWF or Broadcast Wave Format is the European adaption of Microsoft’s WAV and contains embedded metadata in extension chunks, which makes it more desirable as a preservation option. The International Association of Sound and Audiovisual Archives and Audio Engineering Society recommend BWF for archival use [13]. AIFF or Audio Interchange File Format is another uncompressed option but is not as widely used as WAV/BWF [14]. More details about BWF are provided later in this paper. Other audio recordings on analog tape at SIA have been digitized as WAVs, and audio tracks from CDs are ripped as WAVs. Each DAT creates one WAV file ranging in size from 28 MB to 1.2 GB. Audacity has been used successfully with past projects at SIA. It also is free and open-source. More robust proprietary software has been considered and may be acquired in the future when funds are available. Smithsonian Productions recorded many special events, concerts, and Smithsonian Jazz Masterworks Orchestra performances. The original recordings were used to create radio productions such as Folk Masters, Jazz Smithsonian, and Musica de las Americas. In many cases, SIA has the original recordings, the final radio programs, and all of the elements created in between. Smithsonian Productions closed in 2002, and its analog and digital files were accessioned by SIA. The full-scale transfer started with SIA Accession 04-062, Smithsonian Productions, Productions, 1991-2001 [15], which contains DATs and other audio carriers such as Hi8 tapes used for sound recordings. These DATs document production of the Jazz Smithsonian radio program hosted by the late Lena Horne. The program ran from 1994-2001 and featured music from Smithsonian Jazz Masterworks Orchestra concerts recorded by Smithsonian Productions. This accession contains original master recordings, safety copies, and production masters, which means some duplication across recordings. The first box contained 46 tapes, which amounted to 31.44 GB of storage. Even though Audacity is easy to use and does not require sound engineering skills, getting the correct sample and export format settings within it took some additional research and trial and error before successful files were created. Early transfers either caused Audacity to crash or distortion was present on the WAV file. Audacity also had a tendency to crash when other applications were open on the Mac. Even though the accepted preservation standard for audio is 24-bit, 96 kHz [16], this is unattainable with the DATs because they were recorded at 16-bit, 44.1 or 48 kHz, and accurate copies cannot be created at higher sample rates. Sampling at a higher rate could even cause noise within the WAV file [17]. Original rates should be maintained for digital transfers. These DATs typically run 60 to 90-plus minutes, and transfers are done in real time. Only two to three DATs can be transferred and checked for a first round of quality assurance in a workday with SIA’s current procedures. Audacity settings include: Stereo Default Sample Rate – 44.1 kHz or 48 kHz Default Sample Format – 32-bit float (optimal for input even though export is 16-bit) Real-time Dither – None High-quality Dither – None Uncompressed Export Format – WAV (Microsoft) Signed 16-bit PCM (linear pulse code modulation) The pilot resulted in the transfer of 87 DATs of more than 89 hours of audio, which included additional Jazz Smithsonian material and recordings for Smithsonian Folkways’ CDs of Native American music. These WAVs totaled 57 GB. Only four tapes had to be re-transferred due to distortion/noise on the WAV files that was not present in the original recordings. Audio quality was usually good as well in terms of little static or distortion on the DATs. Nevertheless, in some cases, spoken word was low sometimes due to the speakers who were not near the microphone. Edited program masters and air copies did not have this problem. SIA’s best practices strive to handle the original file/object once to create an optimal digital preservation file with the highest standards as possible. From there, derivative use copies can be created that are smaller in size due to compression, such as MP3 files for the web. Due to DAT’s obsolescence issue and the risk of tapes themselves deteriorating, SIA’s goal is to transfer the tape only once if possible. As noted in some instances with distortion, a tape has to be transferred again after quality control is done on the WAV. The transfer starts with the archivist listening with professional headphones to the DAT while recording the audio in Audacity. The headphones eliminate external noise and provide better sound quality than the Mac’s internal speakers. The archivist documents any distortions or other issues on the original. Once the tape has finished playing in the DAT recorder, the recording in Audacity is stopped and the file is exported as a WAV. The WAVs are not altered to eliminate problems heard on the DAT. The WAV file is played back by the processing archivist on the Mac in iTunes as the first quality check. The pilot started out with the entire WAV file being played back but was later modified to sample a few minutes at the beginning, middle, end, and any areas of known distortion by the processing archivist. Each audio file is run against FITS (File Information Tool Set) [18] to validate its format and to generate an MD5 checksum. The XML output from FITs is kept with the accession. Another archivist then performs a second listen of the full file on a PC. Metadata is stored in an Excel spreadsheet. Fields include date of transfer, archivist, file name, start and end times of audio, any issues during transfer or playback, tape manufacturer, and checksum. As stated previously there was some duplication among tapes and this was noted, but no transfers have been deleted as of yet. Files are saved on the Mac, a backed-up preservation server, and LTO tapes. In the future the WAV files could be saved in the Smithsonian’s enterprise Digital Asset Management System once audio is being accepted. Initial research on BWF done during the pilot turned to the work of the Federal Agencies Digitization Guidelines Initiative (FADGI). FADGI’s Audio-Visual Working Group created an open-source free software tool called BWF MetaEdit. BWF MetaEdit can check or add WAV metadata, allowing the BEXT chunk to be populated to create valid BWF files. [19, 20]. A chunk is a self-contained set of data with a header and accompanying data. FADGI recommends the BEXT or Broadcast Audio Extension chunk to include the following elements: Originator, OriginatorReference, Description, OriginationDate, Version, and Reserved. Metadata can be imported or exported with the tool or entered directly [21]. See Figure 2. Software that does not support BWF cannot retrieve the BEXT metadata, but the audio can still be played. A BWF file retains the .wav extension. Initial testing of the tool was conducted late in the pilot with good results, except for some crashes. Since then, additional testing has been done, and a set of WAVs has been saved in the BWF format. The BWF MetaEdit tool soon will be fully implemented at SIA. Figure 2. The description element for the BEXT chunk in FADGI’s BWF